Introduction

blablabla

Data description and preprocessing

The datasets used in the below analysis were sourced from www.kaggle.com website ¹. They were created based on several sources including the Bureau of Justice Statistics ² and FBI Uniform Crime Reporting Program ³. The National Prisoner Statistics Program conducted by the Bureau of Justice Statistics has collected data on the number of prisoners in state and federal prison facilities since 1926. It is produced annually on national and state level. Data are sourced from the 50 state departments of correction, the Federal Bureau of Prisons, and until 2001, from the District of Columbia. The UCR Program provides statistics on violent and property crimes. Data are collected annually and are available on national, state and city level. For the purposes of our analysis we are using state-level statistics.

Additionally, we individually collected data on prison expenditures provided by the Bureau of Justice Statistics ⁴ for each state in 2016 which is the lastest data available. Later in the analysis we will use them in order to correlate the expendirutes with the occurence of particular crimes.

UCR

The UCR dataset consist of 15 variables, two of which are the jurisdiction and year of the observation. It provides information about the state population and also about number of violent crimes (murder, manslaughter, rape, robbery, aggravated assault) and property crimes (burglary, larceny, vehicle theft) per state yearly. Detailed definitions of each crimes can be found on UCR Program website.

The crime_reporting_change variable reflects instances when states’ reporting standards changed. The crimes_estimated variable indicates cases where the FBI computes estimates for participating agencies not providing 12 months of complete data for state ⁵.

ucr <- read_csv("data/ucr_by_state.csv")
ucr$year <- as.factor(ucr$year)

colnames(ucr)

##  [1] "jurisdiction"           "year"                  
##  [3] "crime_reporting_change" "crimes_estimated"      
##  [5] "state_population"       "violent_crime_total"   
##  [7] "murder_manslaughter"    "rape_legacy"           
##  [9] "rape_revised"           "robbery"               
## [11] "agg_assault"            "property_crime_total"  
## [13] "burglary"               "larceny"               
## [15] "vehicle_theft"          "X16"                   
## [17] "X17"                    "X18"                   
## [19] "X19"                    "X20"                   
## [21] "X21"

The ucr dataset has a lot of missing values, compared to the other datasets that have none. We dropped the last 6 columns that were completely empty and then we dropped rows consisting of only missing values. It leaves all columns without any missing values apart from “rape_revised” with 612 missing values and “rape_legacy” with 104 missing values.

# removing last 6 columns
ucr <- ucr[, -c(16:21)]
# removing all missing rows
ind <- apply(ucr, 1, function(x) all(is.na(x)))
ucr <- ucr[ !ind, ]
# showing sum of missing values per columns
sapply(ucr, function(x) sum(is.na(x)))

##           jurisdiction                   year crime_reporting_change 
##                      0                      0                      0 
##       crimes_estimated       state_population    violent_crime_total 
##                      0                      0                      0 
##    murder_manslaughter            rape_legacy           rape_revised 
##                      0                    104                    612 
##                robbery            agg_assault   property_crime_total 
##                      0                      0                      0 
##               burglary                larceny          vehicle_theft 
##                      0                      0                      0

As you can see on plot on the left below, in the last two years, 2016 and 2017, there is an additional obervation ie. jurisdiction. Looking at the plot on the right, New York is missing in one year, Puerto Rico is visible in only 3 years. District of Columbia is sometimes renamed as DC, but overall it sums up to all 17 years.

library(viridis)
plot.data1 = ucr %>% group_by(year) %>% count()
ggp1 = ggplot(data = plot.data1, aes(x=year, y=n, fill=year)) + 
  geom_bar(stat = "identity") +
  scale_fill_viridis_d() +
  scale_x_discrete(breaks = as.factor(seq(2001, 2017,2))) +
  theme_minimal() + 
  theme(axis.title.x = element_blank(), 
        axis.title.y = element_blank(),
        legend.position = "none")

plot.data2 = ucr %>% group_by(jurisdiction) %>% count() %>% arrange(n) %>% filter(n<17)
ggp2 = ggplot(data = plot.data2, aes(x=jurisdiction, y=n, fill=jurisdiction)) + 
  geom_bar(stat = "identity") +
  theme_minimal() + 
  scale_fill_viridis_d() +
  theme(axis.title.x = element_blank(), 
        axis.title.y = element_blank(),
        legend.position = "none")

grid.arrange(ggp1, ggp2, ncol = 2)

Based on the above analysis, we decided to rename “DC” to “District of Columbia” and exclude Puerto Rico state.

ucr$jurisdiction[ucr$jurisdiction=="DC"] <- "District of Columbia"
ucr <- ucr %>% filter(jurisdiction!="Puerto Rico")

# interpolation <- data %>%
#   group_by(country) %>%
#   mutate(valueIpol = approx(year, women_part, year, 
#                             method = "linear", rule = 1:2, f = 0, ties = mean)$y)
# i=0
# for (i in seq_along(interpolation$valueIpol)) {
#   if (is.na(interpolation$women_part[i]) == FALSE) 
#     i = i+1
#   else if (is.na(interpolation$women_part[i]) == TRUE) 
#     interpolation$women_part[i] <- interpolation$valueIpol[i]
# }

We also analysed the missing values of variables rape_revised and rape_legacy. Because there are so many missings and they mostly do not occur in the same year, we can’t compare them and that’s why we decided to drop them.

rape_df <- data.frame(year=as.factor(2001:2017))

rape_revised_count <- ucr[!is.na(ucr$rape_revised),] %>% 
                            group_by(year) %>% 
                            count(name="rape_revised_count")
rape_legacy_count <- ucr[!is.na(ucr$rape_legacy),] %>% 
                            group_by(year) %>% 
                            count(name="rape_legacy_count")
rape_df <- left_join(rape_df, rape_revised_count, by="year")
rape_df <- left_join(rape_df, rape_legacy_count, by="year")

Hide data

Show data

kable_f(rape_df)

year	rape_revised_count	rape_legacy_count
2001	NA	51
2002	NA	51
2003	NA	51
2004	NA	51
2005	NA	51
2006	NA	51
2007	NA	51
2008	NA	51
2009	NA	51
2010	NA	51
2011	NA	51
2012	NA	51
2013	51	51
2014	51	51
2015	50	50
2016	51	NA
2017	51	NA

.

ucr$rape_legacy <- NULL
ucr$rape_revised <- NULL

colnames(ucr)

##  [1] "jurisdiction"           "year"                  
##  [3] "crime_reporting_change" "crimes_estimated"      
##  [5] "state_population"       "violent_crime_total"   
##  [7] "murder_manslaughter"    "robbery"               
##  [9] "agg_assault"            "property_crime_total"  
## [11] "burglary"               "larceny"               
## [13] "vehicle_theft"

pl <- vector("list", length = ncol(ucr[,c(5:13)])-1)
colors <- viridis(8)
for(ii in seq_along(pl)){
  .col <- colnames(ucr[,c(5:13)])[-1][ii]
  .p <- ggplot(ucr, aes_string(x=.col, fill="colors[ii]", color="colors[ii]")) + 
          geom_density(alpha=0.3) + 
          scale_fill_manual(values = colors[ii], aesthetics = c("color", "fill")) +
          theme_minimal() +           
          theme(legend.position = "none",
                axis.title.x = element_blank(),
                axis.title.y = element_blank()) +
          labs(title = .col)+ 
          scale_y_continuous(labels = function(x) format(x, scientific = FALSE))+ 
          scale_x_continuous(labels = function(x) format(x, scientific = FALSE))
  
  pl[[ii]] <- .p
}

grid.arrange(grobs=pl, ncol=2)

(…)

Incarcarations in prison

prison <- read_csv("data/prison_custody_by_state.csv")
colnames(prison)

##  [1] "jurisdiction"   "includes_jails" "2001"           "2002"          
##  [5] "2003"           "2004"           "2005"           "2006"          
##  [9] "2007"           "2008"           "2009"           "2010"          
## [13] "2011"           "2012"           "2013"           "2014"          
## [17] "2015"           "2016"

The prison data, compared to ucr is in a panel form, consisting of years as columns. Using long_panel we converted the dataframe so that each row is a different jurisdiction and year.

colnames(prison)[3:18] <- paste0(colnames(prison)[3:18],'1')
prison_panel <- long_panel(prison, begin = 2001, end = 2016, label_location = "beginning", id = "jurisdiction")
names(prison_panel)[names(prison_panel) == "wave"] <- "year"
names(prison_panel)[names(prison_panel) == "1"] <- "prison"
prison_panel$year <- as.factor(prison_panel$year)
kable_f(head(prison_panel))

jurisdiction	year	prison
Alabama	2001	24,741
Alabama	2002	25,100
Alabama	2003	27,614
Alabama	2004	25,635
Alabama	2005	24,315
Alabama	2006	24,103

Prison expenditures

prison_exp_2016 <- read_delim("data/prison_expenditures.csv", ";")

kable_f(head(prison_exp_2016))

State and type of government	prison expenditure
Alabama	722,269
Alaska	338,005
Arizona	1,684,710
Arkansas	595,731
California	15,468,283
Colorado	1,313,103

State area and region

In order to enhance further visualisations, we add an information about state area and region based on R built-in us_states dataset.

library(spData)
library(sf)
us_states_info <- data.frame(jurisdiction = us_states$NAME, 
                             region = us_states$REGION,
                             area_km2 = as.numeric(round(us_states$AREA, 0)))

kable_f(head(us_states_info))

jurisdiction	region	area_km2
Alabama	South	133,709
Arizona	West	295,281
Colorado	West	269,573
Connecticut	Norteast	12,977
Florida	South	151,052
Georgia	South	152,725

Because of the fact that there are two states missing in the us_states_info dataset, we manually added region and land area for Hawaii and Alaska ⁶.

additional_states <- data.frame(jurisdiction = c("Hawaii", "Alaska"),
           region = c("remote", "remote"),
           area_km2 = c(16638, 1481346))

us_states_info <- rbind(us_states_info, additional_states)

Unifying state names

In the prison dataset, District of Columbia is named as Federal and in prison_exp_2016 is named as Washington, D.C., so in order to unify the names we ranamed both to District of Columbia. We also renamed the variable State and type of government to jurisdiction for easier further calculations.

setdiff(prison$jurisdiction %>% unique(), ucr$jurisdiction %>% unique())

## [1] "Federal"

setdiff(prison_exp_2016$`State and type of government` %>% unique(), ucr$jurisdiction %>% unique())

## [1] "Washington, D.C."

setdiff(ucr$jurisdiction %>% unique(), us_states_info$jurisdiction %>% unique())

## character(0)

prison$jurisdiction[prison$jurisdiction=="Federal"] <- "District of Columbia"
names(prison_exp_2016)[names(prison_exp_2016) == "State and type of government"] <- "jurisdiction"
prison_exp_2016$jurisdiction[prison_exp_2016$jurisdiction=="Washington, D.C."] <- "District of Columbia"

Background

The United States has the largest prison population in the world, and the highest per capita incarceration rate. According to 2018 report of the Bureau of Justice Statistics (BJS), nearly 2.2 million adults were imprisoned in America at the end of 2016. That means for every 100,000 people living in the US, about 655 of them were held in prisons and jails. Because of the huge scale of prisoners in the country, also the expenditures on prisons are the highest. According to recent surveys regarding the United States expenditures, spendings on incarceration have increased about three times as fast as spendings on elementary and secondary education during this time period.

Accorindg to Hartney [^hartney],

The incarceration rate in the US is four times the world average. Some individual US states imprison up to six times as many people as do nations of comparable population. The US imprisons the most women in the world. Crime rates do not account for incarceration rates.

(…)

https://www.nccdglobal.org/sites/default/files/publication_pdf/factsheet-us-incarceration.pdf

liczba prisoners:

prison_panel_year <- prison_panel %>% group_by(year) %>% summarise(value = sum(prison))

p <- ggplot(data = prison_panel_year, aes(x = year, y = value, color = year)) +
  geom_point() +
  scale_color_viridis_d() +
  labs(title = "Number of prisoners in the USA per year", x = "Year", y = "Number of prisoners") +
  theme_minimal() +
  theme(legend.position = "none")

ggplotly(p, tooltip = "text")

Average historical violent crimes vs. property crimes per population across states

library(reshape2)
plot.data <- ucr %>% left_join(us_states_info, by="jurisdiction") %>% 
          group_by(region) %>% 
          summarise(violent_crime_total = mean(violent_crime_total),
                    property_crime_total = mean(property_crime_total)) %>%  melt()

## Warning: Column `jurisdiction` joining character vector and factor,
## coercing into character vector

## Using region as id variables

ggplot(data = plot.data, aes(x=region, y=value, fill=variable)) +
   geom_bar(stat="identity", width=.5, position = "fill") +
  theme_minimal() + 
  scale_fill_viridis_d()

Below can be seen maps of US states divided by the severity of a crime, that is violent compared to property crimes.

#create df with mean values across years per state from ucr
ucr_grouped <- ucr %>% 
                  group_by(jurisdiction) %>% 
                  summarise(violent_crime_total = mean(violent_crime_total),
                            property_crime_total = mean(property_crime_total))
#rename variable for merging
names(ucr_grouped)[names(ucr_grouped) == "jurisdiction"] <- "NAME"
#merge grouped ucr and state spatial data
us_states_ucr <- merge(us_states, ucr_grouped, by = "NAME")

#create values per population
us_states_ucr$violent_crime_per_pop <- us_states_ucr$violent_crime_total/us_states_ucr$total_pop_15
us_states_ucr$property_crime_per_pop <- us_states_ucr$property_crime_total/us_states_ucr$total_pop_15

us_states_midwest <- us_states %>% 
                        filter(REGION=="Midwest") %>% 
                        st_union() %>% 
                        cbind(data.frame(REGION="Midwest")) %>% 
                        st_sf()

us_states_norteast <- us_states %>% 
                        filter(REGION=="Norteast") %>% 
                        st_union() %>% 
                        cbind(data.frame(REGION="Norteast")) %>% 
                        st_sf()

us_states_south <- us_states %>% 
                        filter(REGION=="South") %>% 
                        st_union() %>% 
                        cbind(data.frame(REGION="South")) %>% 
                        st_sf()

us_states_west <- us_states %>% 
                        filter(REGION=="West") %>% 
                        st_union() %>% 
                        cbind(data.frame(REGION="West")) %>% 
                        st_sf()

us_states_regions <- rbind(us_states_midwest, us_states_norteast, us_states_south, us_states_west) %>%                                  st_sf()


# create usa map for both crime types
usa1 <- ggplot() +
          geom_sf(data = us_states_ucr, aes(fill = property_crime_per_pop), lwd = 0) +
          geom_sf(data = us_states_regions, aes(color=REGION), alpha=0, size = 1) +
          scale_fill_viridis_c(option = "viridis", trans = "sqrt") +
          scale_color_manual(values = heat.colors(6)[2:5])+
          theme(legend.position = "none") +
          theme_minimal()

usa2 <- ggplot(data = us_states_ucr) +
          geom_sf(data = us_states_ucr, aes(fill = property_crime_per_pop), lwd = 0) +
          geom_sf(data = us_states_regions, aes(color=REGION), alpha=0, size = 1) +
          scale_fill_viridis_c(option = "viridis", trans = "sqrt") +
          scale_color_manual(values = heat.colors(6)[2:5]) +
          theme(legend.position = "none") +
          theme_minimal()

# format main map
usa_all1 <- usa1 + 
              ggtitle("Property crimes per population")+
              theme(legend.position = "right")
usa_all2 <- usa2 + 
              ggtitle("Violent crimes per population")+
              theme(legend.position = "right")
# zoom and format zoomed map of DC
usa_dc1 <- usa1 + 
            coord_sf(xlim = c(-79, -75), ylim = c(38, 40)) +
            guides(fill=FALSE)+
            theme(axis.title = element_blank(), 
                  axis.text  = element_blank(),
                  axis.ticks = element_blank(),
                  legend.position = "none")
usa_dc2 <- usa2 + 
            coord_sf(xlim = c(-79, -75), ylim = c(38, 40)) +
            guides(fill=FALSE)+
            theme(axis.title = element_blank(), 
                  axis.text  = element_blank(),
                  axis.ticks = element_blank(),
                  legend.position = "none")

# combine both plots and add red rectangle around zoomed area
ggp1 <- usa_all1 + 
          annotation_custom(ggplotGrob(usa_dc2), xmin= -80, ymax= 35)+
          geom_rect(aes(xmin = -79, xmax= -75, ymin=38, ymax = 40), size=1, fill=NA, color="red")

ggp2 <- usa_all2 + 
          annotation_custom(ggplotGrob(usa_dc2), xmin= -80, ymax= 35)+
          geom_rect(aes(xmin = -79, xmax= -75, ymin=38, ymax = 40), size=1, fill=NA, color="red")

Property crimes per population

Violent crimes per population

jakis kolejny rozdział

Statistical analysis of the dataset

jaka jest zależność między liczbą więźniów (prison) a wystąpieniami poszczególnych crime na przestrzeni lat (ucr)? czy wzrost uwięzionych zminiejsza odsetek jakiegoś typu przestępstw? czy może jest stały wzrost/spadek przestępstw? (geom line i geom smooth)

Does this significant investment into imprisonment improve public safety? wydatki na więzienia a wystąpienia przestępstw - ogółem i w kategoriach, w roku 2016 (najnowsze dane); source: https://www.bjs.gov/index.cfm?ty=dcdetail&iid=286

jak wygląda liczba uwięzionych na przestrzeni lat? dla całego kraju i dla poszczególnych stanów?

dodatkowe zmienne -> area (ok) - w kodzie -> wydatki na prisons (ok) - w excelu , 2016

-> co poza mapą i bombelkami? - heatmapa -

-> https://www.datanovia.com/en/blog/top-r-color-palettes-to-know-for-great-data-visualization/

Source: https://www.kaggle.com/christophercorrea/prisoners-and-crime-in-united-states#ucr_by_state.csv ↩
Source: https://www.bjs.gov/index.cfm?ty=dcdetail&iid=269 ↩
Source: https://www.ucrdatatool.gov/Search/Crime/State/RunCrimeStatebyState.cfm ↩
Source: https://www.bjs.gov/index.cfm?ty=dcdetail&iid=286 ↩
“For agencies supplying 3 to 11 months of data, the national UCR Program estimates for the missing data by following a standard estimation procedure using the data provided by the agency. If an agency has supplied less than 3 months of data, the FBI computes estimates by using the known crime figures of similar areas within a state and assigning the same proportion of crime volumes to nonreporting agencies.” (cited from https://www.ucrdatatool.gov/faq.cfm)↩
Sources: https://en.wikipedia.org/wiki/Alaska and https://en.wikipedia.org/wiki/Hawaii ↩

Crimes and incarceration in the United States

Ewelina Osowska and Ewa Sobolewska

10-01-2019